Towards Functional Benchmarking of Information Retrieval Models

To evaluate the effectiveness of information retrieval (IR) system, empirical methods (performance benchmarking) are widely used. Although they are useful to evaluate the performance of a system, they are unable to assess its underlying functionality. Recently researchers use logical approach to model IR properties so that inductive evaluation of IR could be performed. This approach is known as functional benchmarking. The aboutness framework has been used for this purpose. Aboutness based functional benchmarking is promising but yet ineffective due to the lack of a holistic view of the evaluation process. To overcome the ineffectiveness of the existing aboutness frameworks, we apply the idea of reasoning about function to IR and introduce a new strategy for IR functional benchmarking, which involves the application of a symbolic and axiomatic method to reason about IR functionality. This strategy consists of three parts, namely definition, modeling and evaluation. To facilitate the unified logical representation of an IR model in definition part and effective reasoning in the modeling part, a threedimensional scale, which can identify the classes of essential IR functionality (representation, matching function, and Iransformation) is proposed in this paper. With this scale, the deficiencies of the existing aboutness frameworks could be overcome.